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PREFACE 

This paper presents two different approaches to performing self-assessments of continuous 
improvement activities. Case Study 1 describes the activities performed by JSC to assess the 
- implementation of continuous improvement efforts at the NASA Center. The JSC approach 
included surveys administered to randomly selected NASA personnel and personal interviews 
with NASA and contractor management personnel. Case Study 2 describes the continuous 
improvement survey performed by the JSC Safety, Reliability, and Quality Assurance 
(SR&QA organization). This survey consisted of a short questionnaire (50 questions) 
administered to all NASA and contractor SR&QA personnel. The questionnaire is based on 
the eight categories of the President's Award for Quality and Productivity Improvement. It is 
designed to objectively determine placement on the TQ benchmark and identify a roadmap for 
improvement. 


INTRODUCTION 

We have been following Continuous Improvement (Cl) principles at JSC for many years, 
although only in the last several years have we recognized our efforts as part of a Cl program. 
Of course, the degree of implementation varied from organization to organization and we had 
never measured how well we were doing within individual organizations or, for that matter, 
Center-wide. After conducting management retreats and consulting with outside sources, we 
decided that an internal self-assessment of our progress would help us baseline our efforts to 
date. The self-assessment survey would provide a benchmark placement based on the 
President's Award criteria; point out the strengths and weaknesses in our Cl implementation; 
allow development of action plans to focus on areas for improvement; and help identify 
communication problems. 

JSC conducted a Center-wide survey in 1991 and established a benchmark based on the 
President's Award criteria. The same survey was conducted in May, 1992 and other 
measurement techniques were added to supplement this survey. The entire survey in 1992 
consisted of several different data gathering tools including the Center-wide survey 
administered to 325 employees; personal interviews with 100% of our top executives; 
organizational questionnaires focusing on specific Cl accomplishments; and surveys of 
managerial employees at four major contractors concerning the impact of JSC's Cl efforts on 
the contractor community. 

While these Center-wide activities were taking place, the JSC SR&QA organization initiated 
their own independent self-assessment activities. This consisted of a Cl survey conducted in 
June 1992 that was administered to all SR&QA personnel (NASA and contractor). Actual 
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participation in the survey was about 90% (A total of 633 respondents out of about 700 
personnel). As with the JSC Center-wide survey, this survey also provided a benchmark based 
on the President's Award criteria. Loral Space Information Systems, the main JSC SR&QA 
contractor, had previously developed a benchmark in 1991 using the same criteria. However, 
only high level contractor managers were used to establish that benchmark. 

With these two independent and simultaneous Cl survey activities, we learned a great deal 
about ourselves and how effective we have been in establishing, communicating, and 
implementing our Cl goals and objectives. 


CASE STUDY 1 - JSC SELF-ASSESSMENT 


BACKGROUND 

Historically, the JSC efforts had focused on R&D productivity initiatives and the Team 
Excellence program. In the Fall of 1989, we conducted the first self-assessment of our quality 
environment. We did this prior to applying for OMB' s Quality Improvement Prototype 
Award; and indeed, we did receive this award in 1990. Shortly after this self-assessment was 
conducted, a number of our organizations (particularly Engineering and SR&QA) became 
extremely interested in quality improvement. Managers started reading publications and 
attending seminars on Total Quality (TQ), which we are now beginning to refer to as 
Continuous Improvement (Cl). In July of 1990, 150 JSC managers attended a W. Edwards 
Deming seminar held in Houston and jointly sponsored by JSC and Loral, our SR&QA 
support contractor. This was the first exposure to Cl for many of our managers and it helped 
us realize that we needed a formal Cl program at JSC. 

In the Summer of 1990, we formed an ad hoc committee composed of JSC managers to 
determine the strategy to set up and implement a Cl program. The committee members 
decided that the program should start at the top level of management, so they initiated Cl 
training for themselves as a first step. After evaluating several potential consultants and 
vendors, they selected The Cumberland Group to help us develop an implementation plan, 
establish a Cl benchmark and structure, and provide the first phase of Cl training. 
Cumberland conducted the first employee Cl survey in February 1991 and also conducted 
executive interviews. A month later, we held a 2-day retreat for our senior executives and 
their deputies. At the retreat, the executives became aware of and committed to a Cl 
philosophy and approach; updated the JSC vision, mission, goals, and objectives; chartered the 
Executive Council which includes all direct reports to the Center Director; and determined our 
benchmark placement based on the President's Award criteria. 

In April 1991, the JSC TQ Steering Committee, which is composed of the Deputy Directors 
from all JSC organizations, was formed and became the "shepherds' of our Center-wide Cl 
implementation in developing our overall policy and strategy. We also established 
subcommittees to oversee the implementation of Cl training and strategic planning, and the 
formation of process improvement teams. By June 1991, we had started the formal 
implementation of our Cl initiative. This included two day training sessions, taught by The 
Cumberland group, for all managers and supervisors; training of future in-house facilitators; 
training in analyzing work processes and in the use of TQ tools; and training of Q+ Team 
personnel. Our Q-l- Teams (we have one team per directorate or program office) are Cl 
initiative partners with management. They are responsible for establishing systems to 
implement Cl activities in their organizations and for helping our employees become more 
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knowledgeable about and involved in our improvement efforts. With our Q+ Teams trained 
and in place, our managers trained, and our facilitators identified, we were ready to move 
forward with our Cl efforts. 

DEVELOPMENT AND ADMINISTRATION OF DATA GATHERING TOOLS 

About one year into the JSC Cl initiative (April-May 1992), we conducted a self-assessment of 
the state of the Center in preparation for a two day retreat for the Steering Committee and our 
Q+ Team Chairs. The goal was to assess the direction that JSC should take in Cl activities 
for the next year. As mentioned in the Introduction, five data gathering tools were used to 
perform the self-assessment: 

1) A 107 question quality survey keyed to the President's Award criteria was 
administered to 325 randomly selected Civil Service employees. This was 
basically the same survey administered to 125 employees in early 1991, although 
15 new questions focusing on strategic planning, goal setting, and empowerment 
were added. 

2) This same quality questionnaire was administered to our 15 Q+ Team Chairs. 

These employees were considered more knowledgeable than most employees about 
our Cl activities because of their involvement in planning and implementing Cl 
programs for their organizations. 

3) The top 18 members of the JSC senior staff were interviewed to determine their 
involvement in and their plans for Cl, and their view of the progress we have made 
to date in Cl. The interviews consisted of 14 open ended questions specifically 
prepared by JSC for these executive interviews. 

4) Each of the Q+ Teams completed a JSC-designed questionnaire on Cl 
accomplishments in their organization. The questionnaire focused on education, 
improvement opportunity identification, continuous improvement implementation, 
and measurement. 

5) We also surveyed managers at four major contractors to gain their input into JSC's 
Cl objectives, barriers, actions, and plans as they relate to the contractor 
community. 

The data gathered from these five tools were first analyzed on a Center-wide basis. Later, 
individual organization results were available to each Directorate/Program Office for their own 
analysis and action planning. 

ANALYSIS OF FINDINGS 

To analyze the results of the self-assessment, we formed five JSC subcommittees with 
members from the JSC TQ Steering Committee and the Q+ Team Chairs. The subcommittees 
and their areas of concentration were 1) Strategic Planning; 2) Leadership, Empowerment, and 
Training; 3) Process Improvement; 4) Measurement and Benchmarking; and 5) 
Customer/Supplier Partnerships. Each subcommittee had access to all five data sources in 
analyzing JSC's performance in the eight categories of the President's Award. The analysis 
performed by each subcommittee included: 
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1) Correlation of survey questions with the executive interview data. 

2) Analysis of Center-wide responses versus responses from Q+ Chairs. 

3) Analysis of statistically significant differences between 1991 and 1992 data. 

4) Analysis of implications of highest and lowest ranked questions. 

5) Identification of strengths and weaknesses in each Award category. 

6) In-depth review of elements at each level in the President's Award criteria against 
JSC's accomplishments to date. 

7) Benchmark placements and rationale for the placement. 

8) After-the-fact comparison with the 1991 benchmark placements which were 
established solely by JSC Senior Staff at their March 1991 retreat. Except for 
Training and Strategic Planning (which were the two major areas of Center-wide 
focus during our initial deployment of TQ in 1991-1992), all the 1992 benchmark 
placements were lower than those assigned in 1991. 

RESULTS OF SURVEYS AND INTERVIEWS 

The Steering Committee and the Q+ Chairs held a retreat in June 1992 to discuss the results 
of their analysis of the survey data. Each subcommittee presented the strengths and 
weaknesses and the benchmark rating for their assigned areas of concentration. They also 
recommended three objectives for the next year based on the identified strengths and 
weaknesses. This retreat resulted in a common understanding of the future Cl direction for 
JSC and identified specific objectives and action plans to focus our Cl activities for the next 
year. Additionally, we formed a separate training advisory subcommittee to continue our 
emphasis in this critical area. 

Following the June retreat, the Steering Committee met with the Executive Council in July 
1992 to finalize the 1992-1993 objectives and actions and to establish priorities. This meeting 
was designed to get the 'buy-in' of the top tier of JSC executives and to ensure that the 
Executive Council was willing to commit the time and resources needed to achieve the 
objectives. We felt the meeting was very successful and provided the opportunity for the 
Executive Council (the 'Owners') and the Steering Committee (the 'Implementors') to discuss 
and then finalize our Cl direction for the coming year. 

NEXT STEPS 

We plan to conduct a comparable self-assessment annually. The quality questionnaire will be 
reviewed to determine if additions or modifications are required. We also plan to incorporate 
Cl questions in the next NASA-wide Culture Survey. Using these survey techniques and any 
new methods that we develop, we will determine our benchmark placements annually and 
assess the effects of our improvement efforts. In the interim, the subcommittees that we 
established will work to accomplish the objectives and actions that were adopted. Individual 
organizations will also analyze their own survey data to assess their TQ implementation efforts 
and to establish their own objectives and action plans. 
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CASE STUDY 2 - SR&QA SELF-ASSESSMENT 
BACKGROUND 

In conjunction with the Center-wide Cl activities, the JSC SR&QA organization initiated our 
own Cl program. At first, many of these activities were focused on the Loral (contractor) 
side We established a Loral TQ Steering Committee in 1990 to guide our efforts. Following 
this, we formed process improvement teams that were primarily composed of contractor 
personnel. Strategic goals were set and teams were established to define objectives for these 
goals We conducted an internal culture survey and also identified teamwork inhibitors via a 
survey that did include NASA SR&QA personnel. All personnel were briefed on the results of 
the culture survey and the teamwork inhibitor survey. 

In 1991, NASA and Loral joined together to form a joint SR&QA TQ Steering Committee. 
The Deming seminar was conducted in the spring of 1991. Through a series of retreats, we 
developed integrated NASA and Loral goals. In the summer of 1991, we established our first 
TQ benchmark placement, based on the President's Award criteria, using inputs from Loral 
management personnel only. Our process improvement team activities continued, but more 
NASA personnel were assigned to the teams. Our training department began training in the 
use of TQ tools, facilitator skills, and team building for all SR&QA personnel. 

By late 1991 to early 1992, we had conducted strategic planning seminars where we defined 
our Mission, Goals, and Values; our improvement objectives were defined; and teamwork 
between NASA and Loral was really emphasized by management. Loral and NASA 
employees were beginning to work as teammates in their day-to-day activities as well as on the 
teams to which they were assigned. We thought the time had arrived to perform an internal 
self-assessment of the progress we had made in our Cl activities 

DEVELOPMENT AND ADMINISTRATION OF DATA GATHERING TOOLS 

We decided to continue to use the President's Award criteria for our self-assessment 
benchmark because we had a previous benchmark using this criteria and the JSC survey was 
also using this criteria. Thus, this would provide several benchmark comparisons. The 
American Productivity and Quality Center (APQC) prepared the initial questionnaire for the 
survey. We changed some of the wording of the questions and added/deleted some questions 
to tailor the questionnaire to our employees needs. The final questionnaire consisted of 50 
questions, 10 of which were designed to obtain demographic data. In addition to the 
demographic questions and the specific questions, we developed codes for each organization to 
allow sorting by organization and to assure anonymity for the employees. The remaining 40 
questions were keyed to the eight categories of the President’s Award criteria with five 
questions per category. These categories are represented by the bars designated as A through 
H on the SR&QA benchmark Chart (See Figure 1, SR&QA Continuous Improvement 
Benchmark) and are defined as: 

A) Management Leadership & Support 

B) Strategic Planning 

C) Customer Satidfaction 

D) Employee Training & Recognition 

E) Employee Empowerment and Teamwork 

F) Continuous Improvement Measurement and Analysis 

G) Continuous Improvement Activities 

H) Quality Productivity Improvement Results 
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The self-assessment survey had three basic purposes: 1) Provide an objective means for 
determining our placement on the benchmark chart, 2) Provide a roadmap for identifying areas 
that need improvement at the working group level, and 3) Provide a baseline for measuring the 
effectiveness of our improvement efforts. Each group of five questions for the eight categories 
was designed to objectively determine our placement with respect to the five status levels of 
the President's Award criteria. In addition, each individual question was carefully designed to 
measure different aspects of each category. For example, the aspects of Category A 
(Management Leadership & Support) are goal-setting, top-down communication, bottom-up 
communication, evaluation, and recognition. Individual questions in Category A address each 
of the aspects. We had decided that the survey would be provided to all SR&QA employees 
rather than a randomly sampling of employees. This included NASA (at JSC, White Sands 
Test Facility, and Downey/Huntington Beach), and contractor employees from Loral (the main 
SR&QA contractor), SIMCO (who operates the SR&QA calibration laboratory), and Webb- 
Murray & Associates (who are responsible JSC industrial safety, test safety, and the Safety 
Learning Center). We realized that a special plan was required to assure that a large 
percentage of employees participated in the survey. So, on a published schedule basis, we 
provided NASA and contractor focal points in the building where the employees worked to 
solicit their responses. Electronic scan sheets and the survey questionnaire were available in 
these locations and the employees were encouraged to come into the room to complete the 
survey. We briefed them on the value of the survey to them and to the SR&QA organization 
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and assured their anonymity. No names were required on the response sheets, groups with 
less than five employees would not be provided separate reports, and management was not 
allowed to see individual responses. This approach resulted in an overwhelming response: 
with slightly over 700 employees, we received 633 responses to the survey. 

While we were planning the collection of survey data, we contracted with the University of 
Houston Clear Lake (UHCL) to develop the database and reporting system for our survey data. 
The President's Award criteria is scored from 1 to 5 points. After discussion with APQC and 
UHCL we decided to use a scoring scale of 1 to 10 (1 representing inadequate 
implementation; 10 representing excellent implementation). These scores were then converted 
to the 1 to 5 scale. This would allow a larger range of responses and provide more granularity 
in our survey results. 

ANALYSIS OF FINDINGS 

We completed the survey and received our reports in July 1992 from the UHCL. Individual 
reports were prepared and distributed for about 55 organizations. Other reports were 
developed by 'rolling up' individual reports into the next higher level of management. Each 
report included a bar graph summarizing the average response for each survey category as well 
as detailed data on the responses to each of the five questions in each category. The responses 
to the 10 demographic questions were summarized only at the top management level for 
NASA and each of the individual contractors. 

As mentioned previously, the objectives for this self-assessment survey were to determine our 
placement on the benchmark and, more importantly, to identify areas for improvement at the 
lowest possible working group level. The first objective was realized with the average 
benchmark scores from the survey (See Figure 1). The second objective was reached by 
carefully analyzing the survey data to identify areas for improvement, develop action plans, 
and implement actions. The joint SR&QA TQ Steering Committee was responsible for 
identifying the top level tasks that required improvement actions (such as benchmarking 
improvements) and for assuring integration of lower level activities. Each organization 
reviewed their own survey data, discussed the results with their employees, and developed 
action plans to initiate improvements in their work area. These improvement ideas were 
shared with their management and with other organizations. 

RESULTS OF SURVEY 

For the overall SR&QA organization, the benchmark placements for each category ranged 
from 2.6 to 3.2 as can be seen in Figure 1. These scores were rather consistent from 
organization to organization. The lowest scores were in category D (Training and 
Recognition) and category G (Continuous Improvement Activities). Analysis of the responses 
to individual questions revealed that the deficiencies in category D were in measuring the 
effectiveness of our training program and in providing timely recognition of individuals and 
teams. Actions are being taken to develop training plans tailored for each employee. These 
individualized plans will address the needs of each employee in skills training, personal 
development training, and Cl training. Currently, training effectiveness is determined through 
course evaluations that are completed immediately following the course. During the coming 
year, we will be implementing follow-up evaluations three to six months after each course. 
Through this method, we can determine the actual impact each course has had in improving 
performance. We have established a unified recognition program that includes NASA and 
contractor personnel. The timeliness of the presentation of recognition awards has been 
greatly increased. 
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The low score in category G was attributed to the lack of documentation and measurement of 
key work processes. A number of these processes have been documented, but they have not 
been made readily available to most employees. In many cases, process measurements have 
not been established. Our process improvement teams are making progress in documenting the 
processes and are beginning to establish meaningful measurements. Training courses on 
individual work processes are also being provided for employees. We have also identified a 
need for benchmarking studies in significant SR&QA processes. A pilot study has already 
been initiated; more studies will follow. 

In addition to these actions, individual work groups are continuing to identify areas for 
improvement at the working level and are developing action plans to improve those areas. The 
SR&QA TQ Steering Committee is coordinating these efforts and provides a forum for sharing 
the ' lessons learned ' by each work group. The Committee also is responsible for developing 
action plans for certain tasks that must be work at the top level of management. 

NEXT STEPS 

The self-assessment survey will be administered to all SR&QA employees every six months for 
about two years. We will review the survey questionnaire and may modify individual 
questions based on this review. However, we will not make drastic changes in the 
questionnaire because we want to maintain a valid relationship to our established baseline. As 
new benchmark data is obtained, we will assess the effects of our Cl improvement activities on 
the benchmark. Benchmark charts mounted on large boards have been placed in the primary 
buildings where NASA and contractor personnel are housed. The current benchmark 
placements are plotted on these charts and subsequent placements will also be plotted. This is 
a positive way of showing the employees that their participation in the self-assessment survey 
is important and to make them aware of the progress that has taken place in our Cl efforts. 

The value of this type of survey will probably decrease after about two years. So we will 
investigate new methods for assessing our Cl performance. The first method that we are 
considering is the use of certified internal auditors to determine our benchmark placement. 


CONCLUSIONS 

Although JSC and the JSC SR&QA organization are using slightly different methods for 
performing self-assessments, we are accomplishing a common goal: measuring our Cl 
performance so that we can focus our resources on critical areas for improvement. Self- 
assessment using employee inputs is a quick, simple, and effective way of obtaining that 
measurement early in the Cl initiative. 


Al.2-8 



